ON THE DYNAMIC PROGRAMMING PRINCIPLE FOR 
CONTROLLED DIFFUSION PROCESSES IN A CYLINDRICAL 

REGION 

^ ; DMITRY B. ROKHLIN 

O 

^ • Abstract. We prove the dynamic programming principle for a class of diffusion 

h-J , processes controlled up to the time of exit from a cylindrical region [0,T) x G. It 

■ is assumed that the functional to be maximized is in the Lagrange form with non- 
negative integrand. Besides this we only adopt the standard assumptions, ensuring 
the existence of a unique strong solution of a stochastic differential equation for 

, the state process. 

o 

ci \ 1. Main result 

Bellman's dynamic programming (or optimality) principle is a fundamental result 
of optimal control. In diffusion setup it connects a stochastic optimal control problem 

■ with a Hamilton- Jacobi-Bellman partial differential equation [13], [11], [18] . 
In this note we prove the validity of this principle for a diffusion process controlled 

^ \ up to the time of exit from a cylindrical region [0, T) x G, where G C M*^ is an open 

^ ■ set. We consider the maximization problem for a functional in the Lagrange form 

■ with nonnegative integrand. This case was also isolated in [9], where the justification 
of the dynamic programming principle was based on the reduction to the problem 

^ I with optimal stopping and then to the case G = M.'^ (as it was done in [12] )• The proof 

presented below is a more direct one. In fact, it is shown that the argumentation of 
[8] can be adapted to the present case. It should be mentioned that we use several 
^ i ideas of the papers [5], [10]. Note also that the case of infinite time horizon is not 

I addressed in the present paper. 

Throughout the paper we work on the canonical probability space. Namely, let fl = 
C := C{[0,T];M.'^) be the Banach space of continuous M*^- valued functions with the 
norm \\u\\ = supo<t<j' 11^(^)1 and let P be the Wiener measure on its Borel a-algebra. 
So, the canonical process Ws{uj) = Us is the standard (i-dimensional Brownian motion 
under P. Denote by F = {^t)o<t<T the minimal augmented filtration (see e.g. [5, 
Chapter 20) generated by the coordinate mappings Wg^oj) = Ws, < s < T. 

Furthermore, for each t G [0,T] we introduce the filtration F* = (^*)o<s<r; where 
^* is generated by the increments W^ — Wu, t < u < v < s\/t and The filtration 
F* represents the information flow if the observation process starts at time t. The 
information contained in (and concerning P-nuU sets) is available at any time. 
We have F° = F. 
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Let U he a separable metric space and assume that U = IJ^i ^ (^) some 
sequence U{n) C U{n + 1) of its subsets. Denote by ^^{n) the set of F*-progressively 
measurable processes with values in U{n) and put =2/* = IJ^^ s^^{n). Given a point 
{t,x) e [0,T] X M" and a control process a G =2/ := consider the stochastic 
differential equation 



dX. 



t,x,a 



b{s, X*'^'", as)ds + a{s, X*'^'", as)dWs, s G [t, T] 



for the state process X*'^'"" with the initial condition X 
We assume that the functions 



t,x,a 
t 



X. 



b:[0,T]xR'^xU^R'^, a: [0, T] x x U ^ R"^ x 

are Borel and satisfy the conditions 

\b{t,x,u) - b{t,y,u)\ + \a{t,x,u) - cr(t,y,u)\ < Kn\x - y\ 

\b{t,x,u)\ + \a{t,x,u)\ < Kn{l + \x\) 

for {t,x) G [0,T] X M"^ and u,v E U{n) with some constants Kn, possibly depending 
on n. Under these assumptions there exists a unique strong solution X*'^'" of (II. ip 
for any a E £^ . We put X*'^'" = x for s G [0, t]. 
Fix an open set G dW^ and put 

r*'^'" = inf {s > t : X*'^'" ^ G} A T. 

We consider the stochastic optimal control problem 



v{t^x) := sup J(t,x,a) := sup E 



a,) ds. 



where / is a nonnegative Borel function on [0,T] x R'^ x U and f{t,-,u) is lower 



semicontinuous on 



Theorem 1. Let 7^*^ 6e the set of all ¥^ -stopping times 9 such that t < 6 < t. Then 
the function v is lower semicontinuous on [0,T] x and 



v(t, x) 



sup inf 
sup sup 

aG.e/* eeT* , 



/(3,Xf'",a,)rf. + t;(e,X*'^'") 



fis,X^ 



t,x,a 



a. 



rfs + t;(e,X^'"'") 



A similar result in a more general setting was proved in [H] (Theorem 2.1). How- 
ever, we do not assume that the functions b, a, f are bounded and continuous in 
control variable and that the domain G is bounded. For the objective functional in 
the Mayer form an assertion close to Theorem 1 was proved in [10] (Theorem 4.7) 
under the assumption that the diffusion term is non-degenerate. We mention also the 
result of [11] (Chapter V, Theorem 2.1), where the dynamic programming principle 
was proved under the assumptions, ensuring the continuity of the value function v. 
Other references can be found in the cited literature. 
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In Section 2 we collect some auxiliary results. None of them is new. Theorem 1 
is proved in Section 3 with the use of the technique of [S] adapted to the exit time 
problem. Similar approach was used in [10] . 



2. AUXILARY RESULTS 

The following lemma states the existence of a continuous modification y*'^'" of the 
multiparameter process X*'^'" (see pj, Theorem 2.1). This fact is a consequence of 
Kolmogorov's continuity theorem |T7] (Theorem 25.2) or [15] (Chapter 4, Theorem 
72). We mention that it was used in [lOj (in the course of the proof of Proposition 
4.3). For reader's convenience, we outline the proof of this result. Some details are 
borrowed from the lectures on stochastic analysis by Timo Seppalainen. 

Lemma 1. Fix a G ^ . There exists an ¥-adapted process such that 

(i) F*''^'"(co') G C and the mappings 

{t,x) ^ y:*'^'°(tj) : [0,T] xR^^C 

are continuous for each w G fi; 

(ii) ||y*'^'"(w) -X.*'^'"(w)||c = P-a.s. for all {t,x) G [0,T] x R'^. 

Proof. Denote by X*'^'" the solution of (11. ip with the initial condition X^*'^'" = ^ G 
I/^^(t^t) (note that for random ^ we do not define the process X^^'" for u < t). For 
brevity let's omit index a till the end of the proof. As is known (see [T3|, Corollary 
II. 5. 10 and Theorem II. 5. 9), for q > 1 the following inequalities hold true: 

E sup |X*'« - <K{s - ty{l + E|eP«), (2.1) 

t<u<s 

E sup iX'-^-X^'^p" <irE|e-?7p'^. (2.2) 

t<u<T 

Let s > t. From the inequality 

iX^'y - X*'1 < IX^'^ - X*'^| + |X*'^ - X*'1 = \y- X*'^|/[,,](«) 
+ \X:'y - X:-^'V[,,r](n) + la; - y\I[o,t]{u) + - X*'-|/[,t](w), 

where the flow property X*'^ = X^'^"'^ , u > s (P-a.s.) is used (see e.g. [5], Remark 
2.5(i)), it follows that 

sup iX^'^-Xf |2''<42^-if sup \y-X'J\'^+ 

0<u<T \t<u<s 

+ sup |x:-^-X:'^"Y'^ + 1^-1/1"^+ sup |X*'^-Xf l^^y 

s<u<T t<u<T J 
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Estimating according to f l2.ip . (12. 2p the expectations of the terms on the right-hand 
side of the last inequahty: 

E sup \y - X*'^|25 < K{s - ty{l + Ivl^") 

t<u<s 

E sup \X^'y - X^'^^'Y'' < KE\y - X*'^^^ < K^{s - + ly]^") 

s<u<t 

E sup |X*'J' - X*'^f < iriy - a;p^ 

t<u<T 

and interchanging s and t, we get 

E sup \x:'y-Xl;T<K'i\s-t\''{l + \y\"') + \x-y\"^) 

0<u<T 

< K'^(\s - t\' + \y - x\Y' (2.3) 

for (t,x) e [0,T] X (rzA), where A is the hypercube [—1, l]"^ in R'^. 

Put Dk = Tr^TL^^^ and denote by -D = U^^Da: the set of dyadic rational points in 
R^^^. The estimate (ESD implies that the restriction of X.*'^ to ([0,T] x (riA)) fl D 
is a.s. uniformly continuous as a function with values in C (see [15], Chap. 4, the 
proof of Theorem 72). Thus, there exists a set C P(^^') = 1 such that the 
function 

(t, a;) ^ X.*'^(a;) : ([0, T] x M'^) n D ^ C, G 

is uniformly continuous on compact sets. This function is uniquely extendable to the 
continuous function F*'''(cu) on [0, T] x R^ . For cu ^ fi' we put X,*'^(w) = x, s G [0, T]. 

It remains to check (ii) for (t, x) ^ D. Take a sequence {tn-,Xn) {t,^), {tn,Xn) G 
([0,T] X R'^) nL>. We have 

P(||X*'^ - > e) < P(||X*'^ - X*-^"||c > £/3) 

+ P(||X*"'^" - > e/3) + P(||F*-^" - > £/3) 

As ?7. — oo, the first term on the right vanishes by the Chebyshev inequality and 
(12. 3p . the third term vanishes by the continuity of F*'^ and the second term is equal 
to zero by the definition of F*'^. It follows that P(||X*'^ - > 0) = 0. □ 

For X G C and an open set G cMf^ put 

tg{X) = inf{0 <t<T: X{t) ^ G} AT. 

The following result is of folklore type. See however [2j (Lemma 4.2.2) for the case 
of infinite time horizon. 

Lemma 2. The function tq '■ C ^ R is lower semicontinuous in the norm topology 
ofC. 

Proof. We show that the set La = {X G G : tg{X) > a} is open in G for all 
a G M. Note that = C for a < 0. Let X e La with a > 0. Then X{t) G G for all 
t in the compact interval [0, a] C [0,T]. Denote by the compliment of G in M'^. 
Since the distance function 

p{x,A) = mi{\x-y\:yeA}, AcW^ 
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is continuous in a; G M"^: 

\pix,A) - p{y,A)\ <\x-y\ 
(see [1], Theorem 3.16) and p{X(t),G'^) > 0, t G [0,a], we have 

inf p(X(t),G") = 6>0. 

te[0,a] 

Consider the neighborhood of X 

U = {Y eC: sup \Y{t) - X{t)\ < 6/2}. 

te[o,T] 

The inequahty 

p{Y{t), G") > p{X{t), G') - \X{t) - Y{t)\ > 6-6/2 = 6/2, t G [0, a] 
shows that tg{Y) > a for all Y e U. D 

Lemma 3. Fix a G The function {t,x) i— > r*'^'"((X') is lower semicontinuous on 
[0,T] X for P-a.e. co eQ. 

Proof Note that for fixed {t, x) the process (X*'^'")o<s<t and its continuous modifi- 
cation (Y^'^'°')o<s<T, constructed in Lemma [T], are indistinguishable. Hence, r*'^'"(u;) 
is P-a.s. a superposition of the continuous function 

{t,x) ^ : [0,T] xR'^^G 

and the lower semicontinuous function Tg : C h-> [0, T]. □ 

Lemma 4. For any a E the function 

{t, x) ^ J{t, X, a) : [0, T] X [0, +oo] 

is lower semicontinuous. 

Proof Put Bsit, x) = {{s, G [0, T] X R'^ : |s - + \y- < 5^} and denote by 

J^it,x,a) = lim inf J{s,y,a), 

the lower semicontinuous envelope of J. For {t,x) G [0,T] x R"' take a sequence 
(tnyXn) G Bi/n{t,x) such that 

J*(t, a;,a) = lim J(t„, a). 

n—^oo 

By the Fatou lemma we have 

J,(t,x,a) > Eliminf / /(s, X*-^"''", a,) rfs. 

Furthermore, by the (P-a.s.) lower semicontinuity of the function {t,x) i— )■ r*'^'"(a;) 
for any e > the exists N{u) such that 

r*-^-"(a;) > r*'^'"(a;) - £, n> N{u). 

Since / is nonnegative it follows that 

J,(t,x,a) > Eliminf / J(t„,^t...a_^)(s)/(s, X*"'^"'", a,) ds. 
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By the continuity of (t, x) i-)- Xl'^'°'{oj) and the lower semicontinuity of a; i— )■ f(t, x, a) 
we have 

hm inf {s)f{s, X*"'^"'", as) >/(t,r'.-.--.) hm inf f{s, X*"'^"''^, a,) 

n— >oo n— >oo 



Thus, by the Fatou lemma, 



J4t,x,a)>E [ I(t^rt..,c_^)f{s,Xl''''°',as)ds. 
Jo 

At last, by the monotone convergence theorem we get 



J*(t, > hmE / f{s,Xl''''"',as)ds = J{t,x,a). 

The converse inequality is evident. □ 

For an F-stopping time 6' G T := T^j- define the concatenation operator from 
f2 X f2 to f2 by the formula 

{u uj'){u) = uj{u)I[o^e{L,)]{u) + [uj'iu) - uj'{9iuj)) + uj{9{uj))]I^0^^)^T]iu). 

Furthermore, for any function ^ on Q define the shifted function 

We use the following properties of these objects ([5], Proposition A.l(ii) and Remark 
2.5(ii)). 

Lemma 5. (i) Let C, be an integrahle or nonnegative ^x-'^^O'Surahle random 
variable and 9 E T. Then 



E{C\^e)iuj) = / C''^i(^')dPiuj') for P-a.e. cu e il. 
Jq 

(ii) Let a G ^ , 9 G 7^*^. Then for P-a.e. u E Q we have 

X*'"''^(u; ®g io') = X'^"^'''^"'"^"^'"'"(u;'), « ^ [9{uj),T] for P-a.e. co' G fi. 

Note, that although the assertion (ii) of this lemma is formulated in [5] for a 
nonrandom ^ = s, it is used in the above form. 

3. Proof of Theorem 1 
Step 1. Take G [0, T] x R"', a G ^ and G 7;|^t,..a. We have 



J(t, x,a) = E 



f{s, X*'-'°, as)ds + Ei / f{s, Xf as) ds 



(3.1) 



Using Lemma [5]^ii), we get 

r*'^'-(a;®,a;') = rG(X.*'^'"(a;®ec^')) = (^x.'^"^'''"'"^"^'"''" (u;')) = /H'^^'^^H'-'''" (a;')- 
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Furthermore, since {u i^euj')it A6{uj)) = uj{tA9{uj)) and 6{u{- A6{u)) = 9{uj{-)) (see 
[TB] . Example 4.21.3°), it follows that 

e{uj oo') = 0{uj). 

Hence, by Lemma [5](i), 



(c) 




n J9{uj) 



= J(^M,Xf'"(u;),a^'-). (3.2) 
Put 9 = t. As a*'" G from ([31]), (Q we see that 

J{t,x,a)= / J(t,x,a*''^)c/P(w) < f(t,x). 
in 

Thus, it is possible to pass from s^/^ to s^/ in the definition of the value function: 

v{t,x) = sup J{t,x, a). (3.3) 



The above argumentation is completely analogous to [8] (Remark 5.2). The repre- 
sentation (13. 3p and Lemma H] imply that v is lower semicontinuous. 

Now take a G and ^ G 7;|^t..,.. As a^''^ G ^Z^^"'), by formulas (El]), ([32]) and 
the definition of v it follows that 



J(t, x,a) < E 
Evidently, this yields that 



v(t, x) < sup inf E 



j' fis,X'f''^,as)ds + vie,X'/'''] 



(3.4) 



^. Put B{t,x;r) = {{t',x') G [0,T] x : t' G (t - r,t], \x' - x\ < r} 
and fix e > 0. For each {t,x) G [0,T] x R"^ take an e-optimal control a^''^^ G 
f (if:, x) —e < J(t, X, a*'^). For t G [0, T] and a nonnegative continuous function ip < v 
there exist r*'^ > such that 

(p{t, x)+e> (f{t', x'), J{t, X, a*'"^) - e< J{t', x', a^'"") 

for (t',x') G B(t,x;r^'^) since (t',x') h-)- J (t' , x' , a^'^) is lower semicontinuous by 
Lemma m It follows that 

J(t', x', a*'"^) > J(t, X, a*'^) - £ > t;(t, x) - 2e > ^(t, x) - 2e > (f{t', x') - 3e (3.5) 

for {t',x') G 5()f:,x;r*'^). 

Consider on M the upper limit topology with the basis (a, b], a, 6 G M. Under this 
topology R is Lindelof: see [TT| (Chapter 4, §30, Example 3) for the similar case of 
the lower limit topology. Since M'^ under the standard topology is cr-compact, the 
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space M X R'^ is Lindelof under the product topology (see: [3], Fig.l) with the basis 
formed by the sets 

{(t', x') e M X M'^ : t' G (t - r, t], \x' - x\ < r}. 

In this topology the set [0, T] x Mf^ is closed and hence, it is Lindelof in the subspace 
topology with the basis B{t, x; r), {t, x) G [0, T] x R'^. 

Thus, we can take a countable cover {B(ti,Xi; rj))jgpj, = r*''^' of the set [0,T] x 
R'^. The family of disjoint sets (y4j)jgN, defined by 



Ai= B{ti,xi;n), A = B{ti,Xi;ri)\\^^A^j, i>2, 

also covers [0, T] x R'^. For a e and 6 G 7^|^t,^,c put 

= as/[o,e)(s) + ^ «Ua,(6', Xg''''")/[0,T](s), where 



It is easy to see that f3 is F*-progressively measurable. Note that a* G i^/*' C s^^^'^^ 
if (^,X*'''''')(w) G Ai. By ([3IID, and the definition of /3, we have 



Jit, X, 13) = E 
= E 



y|"^(s,Xf'^/3,)d. + J(^,X*'-'^/3^'•) 



Furthermore, by (13. 5p . 

5^ J(^^,Xf '",a^)J^.(^,X*'^'") > 5^(y,(0,X*'^'") -35)/^.(^,X*'^" 



It follows that 

v{t,x) > J{t,x,l3) > E 



j' f{s,X'/''^,a,)ds + v{9,Xl'^n 



-3e. 



Since v is lower semicontinuous, there exists a non-decreasing sequence (v5n)$?Li of 
nonnegative continuous functions convergent pointwise to v on [0,T] x M"^ (see |6], 
Theorem 1.3.7). By the monotone convergence theorem we get the inequality 



v{t,x) > E 



j' f{s,X'f^^,as)ds + v{e,Xf'^) 



-3e. 



By the arbitrariness of ^, a and e it follows that 



^'/(s,Xf'",«,)rfs + t;(^,X, 
Together with (13. 4p this proves the assertion of Theorem 1. 



v{t, x) > sup sup E 
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